feat: versioned DatasetStorage for reading/writing datasets across protocol versions by koenvanderveen · Pull Request #9443 · OpenMined/PySyft

koenvanderveen · 2026-07-15T12:56:52Z

Summary

Mirrors the syft-job migration/storage architecture (PR #9434) in syft-datasets, adding a versioned read/write layer so datasets can evolve their on-disk format while staying compatible with older peers.

DatasetStorage — the single choke point for all dataset filesystem IO. Reads select a ProtocolCodec by the on-disk protocol version and upgrade the object to the latest version in memory; writes downgrade to the target protocol and let the codec persist it. Nests a MigrationRegistry, peer_schemas, and codecs.
v<n> path segment — new datasets can live under public/syft_datasets/v<n>/<name>/ (and private/…). Protocol 0 = the last release (syft-client 0.1.117 / syft-dataset 0.1.20): flat layout, no v<n>, no canonical_name/version fields — read and written back byte-compatibly.
Versioned models — Dataset / PrivateDatasetConfig are now MigratableObjects under models/<object>/v1.py. Object schemas are unchanged since 0.1.117, so only v1 exists; the protocol bump is purely a path-layout + identity-field change. dataset.py is kept as a re-export shim.
DatasetRef — path-derived identity (owner, name, protocol_version); the dataset name comes from the path, not the spoofable file.
Broadcast write — a public dataset is a single copy read by the whole audience (unlike jobs, which target one peer), so the write protocol(s) come from the audience's versions (target_protocol_versions_for_peers), defaulting to the widest-compatible (oldest) protocol when no peers are known. peer_schemas is injectable but not yet populated by syft-client, so today datasets are written in protocol 0 and the v1 layout ships dormant (writing v1 now would hide datasets from 0.1.117 peers).
SyftDatasetManager routes all reads/writes through DatasetStorage and supports multi-version write (one on-disk copy per protocol version in the audience); get_all dedupes copies across layouts, preferring the newest protocol.
Release tooling — scripts/export_release_artifact.py + generate_release_fixture.py; the protocol-0 / 0.1.20 artifacts and fixtures are derived as if that release had emitted them.
Tests mirror syft-job/tests/migrations (unit + p2p), including a byte-exact cross-release round-trip over hand-generated fixtures and a multi-version-write test. CI (unit-tests.yml dataset-tests job) and justfile (test-unit-datasets) wired.

Scope

Package core only, per discussion. syft-client wiring (advertising each peer's dataset protocol version and writing in it at create/share) is a follow-up — syft-client currently exchanges only a single client-version string, not per-package schemas.

Testing

uv run pytest packages/syft-datasets/tests → 40 passed
just test-unit-fast → 406 passed
tests/unit dataset + sync_manager + delete suites → green
pre-commit run --all-files → clean

…otocol versions Mirror the syft-job migration/storage architecture in syft-datasets: - DatasetStorage: single choke point for all dataset filesystem IO. Reads select a ProtocolCodec by on-disk protocol version and upgrade to the latest object version in memory; writes downgrade to the target protocol and the codec persists it. Nests a MigrationRegistry, peer_schemas, and codecs. - On-disk protocol version: new datasets can live under a v<n> path segment (public/syft_datasets/v<n>/<name>/). Protocol 0 = the last release (syft-client 0.1.117 / syft-dataset 0.1.20): flat layout, no v<n>, no canonical_name/version fields; read + written back byte-compatibly. - Versioned models: Dataset / PrivateDatasetConfig are now MigratableObjects under models/<object>/v1.py. Object schemas are unchanged since 0.1.117, so only v1 exists; the protocol bump is purely path-layout + identity fields. - DatasetRef: path-derived identity (owner, name, protocol_version). - Broadcast write: a public dataset is one copy read by the whole audience, so the write protocol(s) come from the audience's versions (target_protocol_versions_for_peers), defaulting to the widest-compatible (oldest) protocol when no peers are known. peer_schemas is injectable but not yet populated by syft-client, so today datasets are written in protocol 0 and the v1 layout ships dormant. - SyftDatasetManager routes all reads/writes through DatasetStorage and supports multi-version write (one copy per protocol version in the audience); get_all dedupes copies across layouts preferring the newest protocol. - migrations/history + release scripts (export_release_artifact.py, generate_release_fixture.py); protocol-0/0.1.20 artifacts and fixtures derived as if that release had emitted them. - Tests mirror syft-job/tests/migrations (unit + p2p, incl. byte-exact cross-release round-trip and a multi-version-write test); CI + justfile wired. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The enclave inference example computed private dataset dirs using the default protocol version (v1 segment), but datasets shared with 'any' are written at the widest-compatible protocol (0, no segment). This mismatch broke the log-sink path and the ensure_logs_dataset idempotency check. Resolve the actual on-disk ref via DatasetStorage, falling back to the widest-compatible protocol for datasets not yet synced.

package-artifacts/ and protocols/ hold released history only. Only 0.1.20 (protocol 0) shipped to PyPI, so the frozen 0.1.21/protocol-1 artifacts were premature. The unreleased protocol 1 already lives in the live registry and will be frozen by export_release_artifact.py at release time (mirrors syft-job). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pjwerneck

A few issues found and commented on.

Overall, the high-level design is in line with #9434, but the fact peer_schemas is still not properly wired is a bit concerning. In production, everything still resolves to v0. I'd be more comfortable if v1 and the whole protocol mechanics was fully integrated and tested end-to-end before wiring it to the rest of syft-client.

DatasetStorage now owns materializing a dataset on disk (the multi-version broadcast write loop + file copying), so the client layer no longer needs to know per-version on-disk layout. - Add DatasetSourceFiles (unversioned DTO) as the create input; storage builds the per-version Dataset objects (URLs are layout-dependent, so a caller can't hand in a finished Dataset). - Add DatasetStorage.create_dataset (multi-version write, single shared identity across copies) and migrate_dataset (rewrite an existing dataset into another protocol layout, preserving uid/timestamps), sharing a private _materialize_version core. File-copy helpers move into storage as _copy_mock_data/_copy_private_data/_copy_readme; copy_dir_contents gains exclude_names. - SyftDatasetManager.create delegates to storage and applies permissions per written copy; add a migrate() wrapper; fix users bare-string char-split. - Add v0->v1 migrate round-trip test; byte-exact protocol-0 round-trip preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DatasetStorage passes the ref when reading/creating a dataset; the model's owner property now derives from _ref.owner (was mock_url.host) and private_dir uses _ref directly. _ref is a private attr with no default. Also drop the dead _validate_dataset_name from SyftDatasetManager (never called; storage's validate_dataset_name still runs via create_dataset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DatasetStorage.create_dataset (and SyftDatasetManager.create) accept an optional protocol_versions list; when provided, the dataset is written in exactly those versions instead of inferring them from the audience. Includes a test and a generate_release_fixture.py tweak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DatasetConfigV1 no longer hardcodes protocol_segment: public/private_dataset_dir derive the v<n> segment from ref.protocol_version (raising if the version has no segment), so a single codec serves every version in protocol_versions. Renamed its iteration to iter_dataset_refs_all_supported_protocols (loops all supported protocols); ProtocolCodecV1 overrides iter_dataset_refs to call it, leaving v0 and the base untouched. A protocol bump without a layout change is now just appending to protocol_versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions Bot added feature pkg:syft-dataset labels Jul 15, 2026